Evaluation plan for the North- and South-Dutch Benchmark Evaluation of Speech recognition Technology (N-Best 2008)

In 2005, the Flemish/Dutch programme STEVIN supported the project N-Best which has the goal to evaluate the performance of present-day large vocabulary continuous speech recognition (LVCSR) systems for the Dutch language. For a number of years, several research groups in Flanders and the Netherlands have studied various aspects of LVCSR systems for Dutch. However, sofar is is hard to compare the results of Dutch LVCSR systems because there is no common evaluation material defined. The N-Best Evaluation is the first attempt to define an evaluation task and speech database and benchmark several speech recognition systems for the performance on this task. The goal is to set up an evaluation framework so that is easier to organize future evaluations, in order to follow system progress and to possibly shift the focus of the task. This document will specify the tasks, data, performance measures, rules and time schedule of the evaluation. The document was first conceived in August 2006 and finalized in January 2007, after consultation and discussion with the Automatic Speech Recognition (ASR) partners in the N-Best project. The evaluation is set up in a way that resembles the various successful NIST evaluations of speech technology, specifically those on speech recognition. We have consulted the various evaluation plans of past NIST evaluations [2, 3], and also the evaluation plan of the French ESTER evaluation [1] held within the EVALDA evaluation campaign. The evaluation will be conducted by TNO Human Factors in the Netherlands (the evaluator), and is open to all research institutes and industries on a voluntary basis. Organizations participating in the evaluation will be named ‘ASR sites’ hereafter. The evaluation will consist of an optional ‘dry-run,’ where the development test data will be formatted in evaluation-style, as well as ASR results, such that both ASR sites and evaluator can ‘rehearse’ the actions needed for running the evaluation, and comment on and possibly alter some aspects of the evaluation. Then, the evaluation will take place by sending data to the ASR site, who will process the evaluation data and send results to the evaluator. The evaluator scores the results after which there will be an adjudication period in which ASR sites will be given the opportunity to comment on certain interpretations and decisions made by the evaluator. Finally, in a workshop the evaluator and ASR sites will present their results to the others accompanied with a paper describing their systems in more detail.